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Criteria sets are a necessary step inj^the systematic 
opment of evaluation in education, Evalu*ation reifeults frojn the 
nation of criteria and evidenced There is a need)):o develop 
cit tools for evaluating criteri/^r ?imilar to those used in 
evaluating evidence. iThe formulation; of 6uch criteria depends on 
distinguishing >etween terms (concepts) and statements (linkages y 
among concepts) . For both terms and statements a further jdistijict ioii 
must be made between categories (nonvariable classes orycittrj^butes) 
and continua (variable continuous dimensions) . Bcnrrowing from the 
d^|.scipline of . sociology^ an exploration fo;: pat^goricarl terms for 
constructing a s^tems approach yields tir^ ^Irements: resourcesr 
structuresr integrationsr 'performance^/ audT outputs. Continuous 
dimensions borrowed from fheory^evelopment are scope, p-arsimony, 
precision,., and accuracy.' Combiiii.p^ these dimensions and elements in a 
four by five solution grid provides both descriptive definitions and 
a prescriptive framework. Affixing labels to e^ch cell of the grid 
provides both literal definitions (labels) and theoretical 
dfef inLtions'^ (dimension b]^ element). Dividing the system's elements 
ilnto their philosophical cdinponents yields ontologyr epistemology , 
ethicsr .aesthetics, and psychology. Crossing these philosophical 
eilements in turn with ythe five system elements yields a five by five 

3attfix which, provides 25 possible elements for the investigation of 
riteria for criteria sets. (JE) 
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1 I'NTioDUCTION 

DefiniHons of EvoluoHon have consisfently of late included criteria or stan- 
dards as a necessary or essential element (20), Since this definition is com- 
monly accepted as consensual fact, this investigation attempted to build 
toward a theory of evaluation using this definitional element. This study 
considered questions as: I 

How can you recognize a Igooc) criterion, or a good set of criteria? 
How can you develop a good criteria set? 
Where do good criteria sets come from? and. 
Who con design good crireria'^eits? 

Little is offered in the literature of Evaloation to suggest how criteria are 
developed, or-where thfey come from, let alone the idea of "goodness", 
Worthen (21) reports 2agen*&ral tasks for evaluation suggested from three dif- 
ferent surveys. Of these 25, four can be perceived by this writer as 
associated with criteria sefs. Fourteen seemed r^st applicable to evidence . 
(measurement and stattstics), six to judgments arid four, other activities: . 
disseminating, recommending, providing feedback and managing resources. 
Of the 81 competencies needed to perform these 25 general tasks, 10 fell 
witftin the area of crit^Via, 51 within evidence and 6 within judgments. 
Fourteen lay within ot^ier areas. This suggests either a lack of emphasis of • 
criteria or possibly criteria are less important,. Because the definition of 
criteria-comparoo-to-evidence implies equal importance, this investigator 
assumed criterid had been underemphosized-in the past. This is supported by 
research for, but presently inadequate, theories of setting performance criteria 
in mastery education (2), an area currently popular in education, « The ob- 
jective of this investigation was to: use fruitful descriptions of evaluation as 
a base to explore, explicate and embed within a framework, criteria for 
criteria sets. 

Criteria for criteria sets^ then, are viewed here as a necessary sfep for 
further systematic development of evaluations in general, and witkin Adult 
EducoHon in particular* 
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EVALUATION AS A UNIT FOR INVESTIGATION 

EvaluaHon has evolved definiHonally from an emphasis on measurement to: 
Criteria and Evidence which lead to a Judgment. Pictorially this is viewed as: 



CRITERIA 






J 






1 


EVIDENCE 





JUDOWS 



These three "boXes" are viewed as elements of a single system— Evaluation. 
These three elements provide a descriptive definition, but because they are 
discrete classes^ provide onty a partial set of tools to perform analyses, ex- 
planations, and predictionso To move beyond description (a fruitful and 
necessary stage) continuous terms and statements are also necessary (6, p. 173). 
This point of necessity raised the questions of sufficiency as well . This in- 
vestigation was not only interested in the necessary conditions for developing 
a theory of evaluation (of which high quality criteria are only a part), but 
also in the sufficient conditions.. Hage (6, p. 172) has suggested a set of 
necessary and sufficient conditions for theory development in Sociology: 

... a theory should contain not only concepts and state- 
ments but definitions— both theoretical and operational — 
and linkages, again, both theoretical and operational. 

To date this set of conditions for theory development has been extremely fruitful 
when applied by this investigator to evaluation systems (11) and to educational 
systems (12, 13). 
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This set of six conditions (concepts, statements, theoretical definitions, 
operational definitions/ theoretical linkages and operational linkages), suggests 
a design that explores and explicates each of the elements in this necessary 
and sufficient set. This would appear as the next step in developing q theory 
of evaluation. 

\ 

Current evaluatipns, as viewed by this writer, are Concerned with only a sub- 
set of the system elements suggested here*. This investigation assumes thatia 
continuously productive approach hinges on a more i$ff active design strategy — 
orfe that considers a complete set of necessary and sufficient conditions. 

STRATEGIES DESIGNED TO DEVELOP BRO^D PERSPECTIVES 

The strategies discussed here have been gleaned from^Hage (6, 8), Nadler 
(17,, 18), Diibin (3) and Johnston (9).* They hciye bel^n previously applied 
successftjWy to evaluation (11) and open learning systems (12, 13). 

Yhe basic design strategy is to first design the formulgfion Intuitively or 
based on whatever scraps of data ore at hand — your fir$t guess-timate Is 
always a free one. \ ^ 

f 

Second, explicate the uhderlymg framework. Third, c^heck evidence with 
criteria, and finally, concentrate on tautologies and inconsistencies within 
the developing framework. 

The first insight provided by this strategy is that each of the three elements in 
the definition (criteria^ evidence, and jucjgments) contains criteria, evidence 
and judgments at a more specific leveL Pictorially this Is displayed as: 



6. 



CRITERIA 



cri teMa 






judgments 










evidence 





EVIDENCE 



JUDGMENTS 



criteria 










judgments 







evidence 



cri teria 









judgments 









evidence 



Within the Evidence box, criteria of evidence (e.g. validity and reliability), 
when compared against the evidence about evidence, allow for conclusions 
about the evidence at hand. In short, one evaluates the evidence. This is " 
done much of the time~explicitly and precisely. Likewise it is assumed 
Criteria, as a unit of investigation, can be evaluated, as can conclusions or 
Judgments. Neither of these investigations seem to be performed explicitly 
at present. 
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Because of a lack of explicit tools to evaluate criteria, and because Criteria 
is seen as an important element of evaluation, tKe remainder of this investiga- 
tion concentrated on Criteria as the unit of inquiry or investigation • An 
explicit set of tools to evaluate Criteria would balance-out the wealth of 
explicit tools presently available and applied to evaluating Evidence. This 
balance would then allow for developing Conclusions systematically, 

A second insight was that diversity (preferably by design) is needed to develop 
broad perspective. In application, this diversity was most fruitfully supplied 
by the diverse languages of logic, mathematics and literature. These distinc- 
tions are explicitly laid out in the Economjc Education Bulletin (4, po 14-15), 

* Logic... has been developed by analyzing, o oOcfual conduct 
of inquiries.,. Application of this theory to qny inquiry 
ordinarily facilitates progress, 

a 

Mathematics, o .serves not only as a meqns of analyzing the - 
connections. » .Tjut also as a means of suggesting other pos- 
sible connections, and therefore of directing the inquiry to 
new data or connections not previously investigated o 

The language (verbal) tool is used as a means of describing^ 
...when properly used, the language tool makes possible 
, • the construction of a word picture that correctly represents 

the steps in the inquiry. 

In this sense, the elements of the necessary and sufficient set form a triad 
of logical, mathematical and literal. 
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Distinguishing between Concepts and Statements (linkages among concepts) 
while retaining each of tHe three languages explicates the necessary and suf- 
ficient elements as: 




OPERATIONAL ^ 

This is a more specific statefnent of the more common definition of a theory: 
concepts linked together. 

These six elements form the necessary and sufficient set based updn the defini- 
tion of theory development stated previously and applied to this incjbiry: 

Each of the-six has a specific purpose or function. 



Literal terms (labels) 
Logical terms (???) 
Mathematical terms ( ???) 
Literal links (connections) 



provide description 

provide meaning 

provide ' measurement 

provide analysis 



Logical links (premises) . provide plausibility 

Mathematical links (equations) provide testability 

The arrows indicate the style of search that has proven most effective for this 
investigator. By definition. Concepts are necessary to form Statements. Con- 
cepts act as TentTS. Then, linked together they form Statements. Given 
labels, search for n^eanings. At this point one can skip frd verbal cbnnections 
and premises without searching for measures. What isn't explicated here is 
how one searches for Terms when starting with StatementSo To date tho scarcity 
of statements in literatures of both evaluation and open learnrng h&s prevented 
the leisure of starting with Statements. 



^0. 



DisHnguishing between concepts and statements is the mbst common definition 
of theory. This definition has been extended here to suggest 3 parts for each 
concept and 3 parts for each statement. j 

Another distinction is implied with this explication of six Elements. Some 
examples within each of the six are categorical classes of^attributes. Other 
exampleys are continuous dimensions or variables. Categories are non-variable, 
Continua are variable. ' ^ 



Adding the distinction between categories ana. continua to the distinction 
between Terms and Statements leads to a fourfmd table: 





TERMS 


LINKAGES 


CATEGORIES 






CONTINUA 







This allows for q third insight. No one cell is more important than any other. 
All four are needed to satisfy this definition of theory development. Thus, all 
four are needed to develop a theory of evaluation which includes criteria for 
criteria sets. With these four possibilities explicated a fourth insight suggested 
that each cell requires techniques to: 

Explore or Search, 

Explicate or specify, and ; 

Embed or structure 



In addition, ways are needed for moving from cell to cell, or understanding 
how the fpur relate together as a single system (cf o 14). . 

Before continuing, an explication of uses for each cell may help clarify 
subsequent applications. 

Categorical terms are used for defining units of inquiry, 
when identifying cutpoints or qualities and when identifying 
elements or major dosses of phenomena. 
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Continuous terms are needed for Theoretical Deflni+ions and 
Operational Definitions* This is crucial. All derived terms 
are continuous^ These are the building blocks of each con- 
tinuous statement. 

Categorical statements ore mandatory for definitional 
premises and for thresholds qlong operational linkages. 

Continuous statements are needed for Theoretical Lmkages 
and for Operational Linkages. 

These four cells suggQst productive uses for each. They do not consider cafe- 
gorical Jinks among continuous terms nor do they consider contini/ous links 
among categorical tenms. They do not mix categorical and continuous modes. 
In general, these mixed modes are unproductive. 

EXPLORING, EXPLICATING AND-EMBEDDING CRITERIA FOR CRITERIA SETS 

Clonceptual Definitions: Literal, Logical, Mathematical 

The unit of inquiry Is Criteria of Crfteria'as an element of evaluation. This 
inquiry is delimite(j to on§ of the nine elements specified in the specific def- 
inition of evaluation. At this point Criteria of Criteria remains -a black box- 
es yet undescribed, meaningless and not measured. 

According to the first cel^, in the explicated fourfold table, exploring for 
categorical terms should yield major classes of phenomena. Initial explora- 
tions were directed toward a^stems approach (it had been fruitful In past 
inquiries). Consequently elements of a system were borrowed from Sociology 
(6, po 234). This is also consistent with Kreltlow/s advice of borrowing from 
a related field (10). The elements borrowed were Resources, Structures , 
Integrations, Performances , and Outputs . Synonyms for these elements are 
Inputs, Arrangements, Articulations, Adequacies and Accumulations. 

These elements provided major classes of possible phenomena operating within , 
the Investigative unit of Criteria of Criteria. So far only labels have been 
suggested. 

Continuous tenms are needed for specifying theoretical definitions, which, 
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provide medning; and operatibnal definitions which provide measurement. 
Again dimensions — variable but Contentless--were borrowed from thebry 
development (6, p. 178) and evaluation (IJ). These dimensions are 
Scope, ^Parsimony, Precision and Accurpcy, 

As originally applied to theory^, p. 178-180), Scope was operational ized 

derived terms 

as a ratio of derived terms to primitive terms or: primitive terms 

In other words, given a certain number of primitive terms the maximum 
number of possible derived terms can be determined using a mathematical 
formula, since derived term? are some combination of the primitiveso The 
" derived 

derived terras in the — r-rri — ratio have a defined upper bound. This 
^ ^ J primitive rr • 

is consistent with thf idea that all inductive conclusions are in some sense 
probable. A priori mathematical pTdbabiFity is defined as the rqtio of 

favorable outcomes This discussion suggests th&t derived terms are simply 

possible outcomes, r ii 

I • r favorable i r. i i . . . 

thg ratio of possib l e or defined by an qpnori mathematical 

"probaBility . This is suggested to explicate the practice of including only 
favorable outcomes in one*s logical derivations o ' 

By combining the idea of a priori mathematical pfobqbility (favorable:possible) 
for djerived terms with the idea of derivedrprimitiye as o. measure of Scope the 
explicit operational definition of Scope becomes: 

favorable ^ 



ssible 



primitive terms . 

This definition indicates that the wider the ratio, the more general the Scope. 
This ratio transfers Scope as a dimensfon out of the domain of raw numbers or 
arriount and into the domain of relational units. 

The theoretical definition of Parsimony is explaining as much as possible with 
as litfl^ as possible. ^Parsimony is operationally defined as d ratio of: 

Pemises^ There is a large but finite set of equations. Again, only equations 

which offer explanation (a favorable outcome) are'retainedo Therefore the 

1. I , r. rn • . -favordble 

explicit operational definition of Parsimony is: possible 

> . i^t premises o . 
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Pcecision fs defined as a ratio unexplaine d Unexplained effects are 

considered td be error. Another way to state this definition is as a ratio of 
explicit effects y^jg suggestion explicates increased Precision. If explicit/ 
implicit effects. ' 

or explained, effects are considered as favorable outcomes, and implicit effects, 
or error> are considered as unfavorable then precision operates as a probability 
of reldtive frequency. Relative frequency statements (odds) suggest that a person 
should never have more faith in an outcome than the evidence merits. In this 
ca$^e more precision Ts not necessarily better. Optimum precision Is determined 
by how much foith is needed for any particular Conclusion, 

Precision is applied to either a single equation or to d simultaneous set of 
equations. Operationally, precision is the ratio of explicit effects to error 
(implicit effects), 

'■■■/- \ ■ 

effects Qp favorable 



/ error unfavorable 

More precision either requires adding additional terms to the equation or 
equations to the set (explicating a portion of the error term); or assuming a 
priori, zero coefficients among sqme of the dependent variables (assuming null 
relationships). This is pointed out in Bidlock (I, p. 59-65) as part of the 
problem of identifying systems of equationSo Morrison (16, p. 53) states that 
a system is inconsistent if it doesn*t meet minimum precision » These two refer- 
, ences indicate that a lower limit on precision is the limit needed to just identify 
a system or to form a consistent system. 

Accuracy applies to the explanations of a theory. This dimension lies withiri 
the set of premises. Accuracy addresses the question: which set of premises 
is most accurate when several available sets appear equally plausibleo Oper- 
ationally this is the ratio of: statement 

all other statements. 



This is a case of the Probability of Confirmation. The conclusion does not fol- 
low necessarily, but only to a particular degreeo This ratio is synonymous 
with plausibilityo . 



Additional accuracy requires either Agreement or Differences as a strategic 
test. An example for the method of agreement Is: 

■ 

\ • BCD do not occur in all 3 left hmd ' 

^gCD ^ g terms. Therefore, BCD are not con- 

^gPQ y E sidered necessary to generate E. 

y^CQK ^ £ A, on the other hand, Occurs in all 

3 left han^l terms. The conclusion is 

thaf A may be necessary. 

1, . ' 
An example of the method pf differences is: 

ABCD — > E . Where A ?^ generates E but not A 

-J - (A)? generates not E (E). The 

AbL — > t conclusion is that A may be sufficient.' 

In either method, agreement or differences, the Probability of A being accu- 
rate in relation to other statements increases even though it has been proven 
to be neither necessary ^jor sufficient. This discussion provides both theoretical 
and operational definitions* for the four borrowed dimensions. 

Dimensions, as defined here, are contir\uoUs but contentless. Content is 
supplied by the Elements defined previously. Elements are categorical 
terms. Multiplying each Dimension with each Element (crossci ossifying) 
yields potential Continuous Terms. These continuous terms specify 
potential theoretical and operational definitions. 

The four dimensions (Scope, Parsimony, Precision, Accuracy) form one 
side of a two-dimensional array. The five borrowed elements (Resources, 
Structures, Integrations, Performances, Outputs) form the second sideo The 
4X5 array that results from the cross-cldlsifi cation provides 20 cells or 
boxes which make-up the initial solution grid or matrix. This strategy 
moves the inquiry from a descriptive definition to a prescritive framework. 
Each cell provides both dirj^nsionality or variability and attributeness or 
contents Each cell th|n 15 a potential Genefol Variable (a variable is 
considered to be general when it is free of places and times). 
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The lack of specified statements (connections, premises ond equations) 
on Evaluation in general and Criteria of Criteria in specific, prevents ^ 
further definition of Parsimony, Precision and Accuracy at, this timie» 

.« 

Labels and Meanings have been tentatively established and these can be 
applied to establish verbal statements and premises. Before continuing to 
that stage hdWev6r, a test of suggested labels and meanings was conducted, 
This was carried out to prevent continued efforts with faulty or incomplete 
Terms. 

DATA 



Test of Names and their Meanings 

A literal definition (label) was inserted into each ce|l of the matrix. This 
provided both a the9retical definition (dimension by element) and a literal 
definition. 



ELEMENTS 



INPUTS 



STRUCTURES 



INTEGRATIONS 



PERFORMANCES 



/ 



ouTPigi^ 



'SCOPE 



Utility 



Generality 



Applicability 



Importance 



Completeness 



PARSIMONY 



Explicitness 



Simplicity 



Matchability 



Necessities 



Consistency 



PRECISION 



Reliability 



Univocality 



Orthogonality 



Objec tivity 



Confidence 



ACCURACY 



Truth 



Validity 



Soundness 



Explanations 



Power 



This 4x5 matrix brings out nine primitive terms (5 elements and 4 dimensions). 
Since the matrix cross-classifies elements by dimensfons the maximum possible 
number of derived terms is 20, assuming null interactions, which is assumed 
for the moment. (During the discussion at St. Louis it was suggested that we 
cross-classify elements by elemenfso This allows for many more primitive terms 
at a more specific level. This will be discussed in a later section of this reporto) 
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sHng of definitions^ both theoretical and literal, provides the following: 



EORETICAL DEFINITIONS 

scope of inputs 
^cope of structures 
scope of integrations 
scope of performances 
scope of outputs 

parsimony of inputs 
parsimony of structures ^ 
parsimony of integrations 
parsimony of performances 
parsimony of outputs 

precision of Inputs 
precision of structures 
precision of integrations 
precision of performances 
precision of outputs 

accuracy of inputs 
accuracy of structures 
accuracy of integrations 
accuracy of perfo|fnances 
accuracy of outpi:^ 



LITERAL DEFINITtONS 

utility- ? 

generality 

applicability 

importance 

com((>leteness 

expllcitness 

simplicity 

moTchability 

nej/cessities 

c<insistency 

reliability 

univocality 

orthogonality 

objectivity 

confidence 

truth 

validity 

soundness 

explanations 

pow^r 



Generally, theoretical definitions are easier to understand and agree to 
than are labels or literal definitions. Eqch ^respondent prefers a particular 
label for each cell. Those suggested here are simply the label which this ' 
investigator believed captured the mbst meaning for him. As such they 
are only one set of many possible sets. This- set of labels wqjs the tentative 
jset designed prior to the St. Louis discussions. 



A completed definitional listing would of course include operational 
definitions or measures. Following the rule that two perspectives are 
sufficient to provide a cross check on each other greater efficiency 
is obtained by comparing each available pair of definitions rather than 
attempting to identify a third perspective for each of the Terms. It was 
with this in mind J-hat the listing of theoretical definitions and literal 
definitions was used as initial input to the Sto Louis discussionso 



The elements of Resources, Structures, Integrations, Performances and 
Outputs are general and apply to any system. More specific elements 
,for a sysfem of Evaluation were borrowed from Morris (15, p, 466), This 
analogy followed the lead given by those who point out the root of the 
word evaluation is valueo Morris lays out five philosophies of western 
man. The-cdmmonalities of these five are the existence of ontology, 
epistemology and axiology as elements. These elements were superimposed 
over the borrowed elements of the more general system to yield: 



RESOURCESf 


STRUCTURES 


INTEGRATIONS 


PERFORMANCES 


OUTPUTS 


ontology 


epistemology 


axiology 


??? 


??? 



The assumed sequence for social systems is that Resources are Structured 
and then Integrated. The assumed sequence for philosophies is that a 
particular view of reality (ontology) impacts upon the way knowledge is 
structured (epistemology). Epistemology in turn limits what is seen as good 
(ethics) and what is beautiful (aesthetics). Ethics and Aesthetics are. con- 
sidered here to be subcategories of Axiology. No philosophical elements 
wer§ identified at this stage for either Performances or Outputs. 



The matrix that results from cross-classifying these six elements with the 
four previously defined dimensions provides a prescriptive shopping list of 
24(4 X 6) possible continuous terras as theoretical statements which may or 
may not apply to Criteria of Criteria. It is so far only a checklist — each 
cell does not necessarily require an entry o As a check on their application, 
theoreticdl definitions are provided by combining each row entry .with each 
column entry. These theoretical definitions are: 



Scope of ontology 
Scope of epistemology 
Scope of ethics 
Scope of aesthetics 
ScopQ of performances 
Scope of outputs 

0 

Parsimony of ontology 
' Parsimony of epistemology 
Parsimony of ethics 
Parsimony of aesthetics 
Parsimony of performances 
Parsimony of outputs 



Precision of ontology 
Precision of epistemology 
Precision of ethics 
Precision of aesthetics . 
Precision of performances 
Precision of outputs 

Accuracy of ontology 
Accuracy of epistemology 
Accuracy of ethics 
Accuracy of aesthetics 
Accuracy of performances 
Accuracy of outputs 



The first test conducted at St. Louis was for literal definitions for each of the 
four suggested dimensions (primitives). Since dimensions are content-less. 
Criteria was considered as the system under discussion. Thus literal defini- 
tions were suggested forlScope of Criteria, etc. This was conducted as a pre- 
sentation work session at the Adult Education Research Conference, St. Louis, 
April 16, 1975. Fifteen of the 32 persons present shared suggested labels o 
Past experience led to the writer's expectations of 4 labels from each respondent; 
12 of the 15 who turned in the worksheets met or exceeded this expectation. 
However, 17 participants chose not to turn in their worksheets. The labels for 
each dimension were ordered according to level of generality and for consis- 
tency with accepted usage (19). In each case generality (the more the better) 
was sought. Labels were included when usage was consensual in the literature 
of theory development. Unacceptable labels we explained as exceptions and 
were not included in further analysesp These labels suggested ore summarized as: 

Scope of Criteria ' 



size (C) 
largeness (C) 
quantity (C) 
extent (2) (C) ' 
specificity (C) 

inclusiveness (4) (I) 
necessary criteria (I) 
part to whole (I) 

cover 96% of situations (G) 
spectrum (G) 
theoretical breadth (G) 
conceptual breadth (G) 
conceptual dimension (G) 
variance of thought or behavior (G) 
comprehensive (G) 
restrictiveness (G) 



focus (B) 
parameter (B) 
boundary (B) 
circumscribe (B) 

complexity (?) 



KEY 



1 . Number in parenthesis denotes 
number of participants listing 
that particular label, if it was 
more than one# 

2. C = completeness 
I = inclusiveness 
G = generality 

B = boundary 

? = non-consensual usage 
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Of the 25 labels suggested: 

6 suggested completeness 
8 suggested generality 
6 suggeste<;| inclusion 
4 suggested boundaries 

1 (cortfiplexity) didn'^t suggest any aspect of scope because scope 
and complexity are considered by this writer to be independent 

JSIp clear cut distinctions appeared among completeness, generality and 
inclusiveness. Any of these 3 (20 responses) portray the idea of scope. 
Boundaries also apply to space rather than abstract relations (19). 

Parsimony of Criteria 

productivity (A) 
adequacy (A) 
scientific (A) 



practical (A) 




KEY 


mathematical (A) 




utilitarian aspect (A) 






necessary and sufficienf A and B (A) 








A = 


Adequacy 


common terminology (S) 


S = 


Simplicity 


simple to under>5tand (S) 


= 


Economy 


easy to explain (S) 


? = 


seemed more in line 






with Scop^ than with 


limitation (?) 




Parsimony ^ 


selection (?) 




_^ — 1^ 



generallzabilify (2) (?) . 

economic (2) (E) 
effective use of resources (E) 
input/output (E) 
product/time (E) 
product/intent (E) 
time (E) 



Of rtie 22 labels suggested: 
3 suggesfed simplicity (S) 

8 suggested careful use of resources or economy (E) 

7 suggested adequacy (A) v 
4,were questionable (?) and appeared to apply to scope rather than parsimony. 

Simplicity appeared to suggest plainness or uncompounded state. Economy 
suggests careful Cind prudent use of resources. Adequacy on the other hand^ 
suggests a sufficiency of the end product. This. provides the often cited ratio 
of inputs/outputs. 



Precision of Criteria 
exactness (E) 

pin pointing (E) « 
on target (E) 
reassurable (E) 
•OS(E) 

sig confidence level (E) 

exemplifying (F) 
usefulness (F) 
directness (F) 

can be 100 at Ot?) ■ > • 

descriptive (?) 

specificity (2) (?) Scope 
congruence of data to criteria ^ . 
validity > Accuracy 

non-overlapping sets of criteria-, n * r% 
V V V V > Parsimony? 
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E == Exactness 

F = Formal n ess 

? = questiond}le meaning 



0 



2b, 



Of the 17 labels suggested: 

6 suggested exactness (E) 
3 suggested formalness (F) * 
2 are questionable (?) 

Specificity, mentioned twice, is more akin to scope as previously defined • 
Non-overlapping sets, mentioned twice, suggests a parsimony dimension • 
Validity and congruence suggests accuracy rather than precision. These con- 
clusions point out the dual themes of exactness and formalness as tapping the 
domain of precision. 

Accuracy of Criteria 

rightness (C) 

correctness (C) KEY 
correctness of fit (C) 
congruent with reality (C) 
validity (C) 

measurement compared to 
actual perfoimance (C) 

appropriateness (?) 
direction (?) 
target (?) 

cannot be interpreted without 
data or a scale (?) 

prediction (P) 

applicability (S) 
exclusiveness (S) 

Of the 15 labels suggested: 
8 suggested correctness (C) 

4 were questionable (?) and not particularly meaningful 



C = correctness 

? = questionable meaning 



Applicability (S) and exclusiveness (S) suggest generality or scope. Predictive^ 
(P) is associated with precision rather than accuracy. These literal definitions 
suggest: 

completeness 

generality and V as components of scope 



inci 



usion 



] 



simplicity 

adequacy and } as components of efficiency 



economy 



} 

} 



exactnes^-and ^ qj components of precision, and 

formal ness 

correcjness as the sole component of accuracy. 

The discussion following the handout suggested three major shortcomings: First 
that the labels were "value-laden". This wrifer has no defense against such a 
suggestion. This objection points directly to an issue of inquiry. Is the goal to 
become value^free or is the goal to become objective? To some this is a trivial 
point because they assume it is impossible to be both value*laden and objective. 
These same people implicitly conclude the way to be objective is to become 
value-free. This is'isuggested as the logic behind the objection to specific labels 
because they are value-laden o 

As the design matrix explicates, a goal of evaluation is objectivity (precision 
of performances). Nowhere to this point has value-freehess been suggested as 
a goal of Criteria of Criteria. Based on this and the resulting objection of 
value-laden labels the crux of the argument centers on whether or not it is 
possible to be both non-neutrol (value-laden) and objective. 

Following the lead suggested by Hage (7), this writer^s answer is a resounding 
Yes— the classical literatures notwithstanding. 

In general, it is assumed that people can either hold d single altruistic goal 

or hold multiple self-interest goalso As portrayed in a 2 x 2 array this suggests . 

either cell 2 or cell 3. 
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GOALS 





SINGLE 


MULTIPLE 


V SELF 
^ I^fTEREST 

L 
U 




2 


E 

S 

ALTRUISTIC 


3 


— 

\ 



The possibility of being both non-neutral and objective lies along the main 
diagonal (cells 1 and 4). Is it possible to move from cell 1 to cell 4 as the 
arrow indicates? Sfiecific values are non-variable concepts. They lead to 
st^eotyped either-or debates, pitting "us" against "them". 

To movfesfrom cell 1 (specific values and single goal) to cell 4 (general values 
and multiply goals) requires both a shift from categorical to continuous thinking, 
and a shift frohissii^^ to multiple goals. 

Both thinking !n contihu^and thinking from multiple perspectives lead toward 
non-nqutral objectivity betquse of the following: 



1. Continuous concepts lead to addpifing an evolutionary rather than, a 
revolutionary perspective. 

2. Continuous concepts force yoii to think in extremes-- then it becomes 
impossible to be an extremist. 

3. A shift to multiple variables is needed to capture all bf reality. 
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4. A shift to a perspective of multiple values prevents ends from justifying 
the means. 

5. Multiple values and multiple variables allow a move toward alternate 
ways of reaching the same endo 

6. Multiple alternative ways and multiple ends approach objectivity 
without neutrality, 

7. As you increase the number of variables or the number of alternative 
pathways the number of Internal checks increase geometrically. 

8. Thus it becomes harder to distort. 

9. Internal consistency allows for non-neutral objectivity. 

The design strategy suggested in this paper addresses the value-laden question 
head-on. A joint attack of variable concepts«and multiple perspectives is 
suggested as a way of penetrating this paradox. On the other hand, this 
objection can certainly be expected since no measures were forthcoming. This 
is not a weakness of the approach but rather a weakness of its specific appli- 
cation here. Second, the question was raised as to Cross-classifying structure 
and function with the elements df ontology, epistemology and axilogy. At 
first glance this suggestion was rejected (structure was based as epistemology) 
and function was part of the system, but had no completed cells. More 
thought however indicated the heuristic value of crossing the philosophical 
elements With the system's elements. This is given By the matrix. 





INPUTS 


STRUCTURES 


INTEGRATE 


PERFORM 


OUTPUTS 


ONTOLOGY 


ontologfcal 
inputs 


ontological 
structures 


ontological 
integrations 


ontological 
performances 


ontological 
outputs 


EPISTEM- 
OLOGY 


epistemol 
inputs 


epistemol 
structures 


epistemol 
integrations 


epistemol 
performances 


epistemol 
outputs 


ETHICS 


ethical 
inputs 


ethical 
structur^ 


ethical 
integrations 


ethical 
performances 


ethicql 
outputs 


AESTHETICS 


aesthetic 
inputs 


aesthetic 
structures 


aesthetic 
integrations 


aesthetic 
performances 


aesthetic 
outputs 


PSYCHOL- 
OGY 


3sychological 
inputs 


psychologica 
structures 


psychological 
integrations 


psychological 
performances 


psychological 
outputs 
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The fifth row (psychology) was added after further discussion about the matrix. 
This suggestion. sprang from the inability to suggest adequacies of performance 
and the historical precedent of Psychology as a branch of Philosophy. As each 
cell is examined the question is asked: 

What are (row)^ (column)? For example^ what are Ontological Inputs (row 1, 
column 1)?' Notice the cell entries are categorical (non-variable) terms. 
This is the case because this matrix is crossing elements with more elements 
which can only yield more categorical terms. These can be appropriately used 
for investigative units^ elements of an investigative unit or modal cut points- 
along a continuum (qualities). The main interest here is identifying elements 
for the investigative unit criteria for criteria sets. This matrix provides 25 
possible elements. Before automatically using all 25 and re-applying the four 
previous criteria of scope, parsimony, accuracy and precision (which would 
yield 100 potential general variables) each element has to "make sense" or 
fit into a logically consistent system. 

The strategic question is simply, which, if any, of the 25 possible elements 
are necessary for defining criteria for criteria sets as an investigative unit? 

One approach is to explicate the definitional premise: Bench marks or 
comparative reference points to judge worth of a set of criteria? 

The third criticWmiOf this design strategy was that perhaps it really doesn't 
portray multiple p^pectives at all — but rather says the same thing in three 
different ways. Since no operational definitions (measures) were developed 
this criticism became one of deciding if literal definitions (labels) were syn- 
onymous with theoretical definitio^^j(meanings). The literature of theory de- 
velopment suggests this is not the casie as do this investigator's experiences. 
As a result of the discussion in St. Louis it is suggested that the distinction 
was blurred in communicating the idea, but the idea of two distinct definitions 
is supported by the discussion. Such comments as "it is easy to see we all 
agree on the theoretical definitions but the labels mean specific things to each 
of us"; or "labels are too value-laden"; or "it seems to me you are merely 
playing with words", suggest the discussants sensed the differences between 
labels and meanings, but failed to move toward verbalizing the differences by 
concentrating either on labels or meaningso No single person appeared to 
attack or support both .conjointly. 



SUMMARY 



As this discussion shaped out, there was a vague uneasy feeling that it was 
not achieving closure. That is true of course, but perhaps irrelevant. For if 
this is truly a dialectic creative-critical effort, this writer's experience sug- 
gests it requires a minimum of two rounds and usually three to achieve tenta- 
tive closure. The discussion during St, Louis AERC completed the first round. 
Your written critique of the points discussed in this /nanu script will complete 
the second. 

Looking back it appears that the first question raised (how can you recognize 
d good criteria set?) still remains open. Tentative answers to remaining 
questions appear more explicit. 

How con you develop a good criteria set? Applying techniques of continuous 
thinking and multiple perspectives has been successful for this investigator. 
To date these ideas hold the most promise of any for gaining scope, parsimony, 
precision and accuracy. 

Where do good Criteria sets come from? .They come from explicit and logical 
techniques of analysis and synthesis. So far that is merely a personal opinion 
of this writer based on personal experiences. Following the systems approach 
there are assumed to be alternative ways to achieve any end. Finally^who 
can design a good criteria set? Anyone can! That includes you. That includes 
me. This follows, providing logical and explicit techniques can be communi- 
cated between people. Once techniques are explicated, and are logical, 
setting criteria can no longer be considered an art. Rather than subtle, 
sophisticated and implicit, the goal is convincing, simple and explicit, 

"Truth emerges more readily from error than from confusion," 

(Francis Bacon) 

" • ft 

Your critique as cycle two of our communication should hopefully point to our 
errors. 
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